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The Total Survey Error (TSE) paradigm provides a framework that supports the effective 
planning of research, guides decision making about data collection and contextualises the 
interpretation and dissemination of findings. TSE also allows researchers to systematically 
evaluate and improve the design and execution of ongoing survey programs and future 
investigations. 

As one of the key aims of a TSE approach is to find a balance between achieving a survey 
with minimal error and a survey that is affordable, it is unlikely that a considerable number of 
enhancements to regular programs of research can be made in a single cycle. From an 
operational perspective, significant alterations to data collection processes and procedures 
have the potential to create more problems than they solve, particularly for large-scale, 
longitudinal or complex projects. Similarly, substantial changes to the research approach can 
have an undesired effect on time series data where it can become difficult to disentangle 
actual change from change due to methodological refinements. 

The University Experience Survey (UES) collects feedback from approximately 100,000 
undergraduate students at Australian universities each year. Based on previous reviews of the 
UES, errors of measurement appeared to make less of a contribution to TSE than the errors of 
representation that were associated with the survey. As part of the 2013 and 2014 
collections, the research design was modified to directly address coverage errors, sampling 
errors and non-response errors. The conceptual and operational approach to mitigating the 
errors of representation, the cost effectiveness of the modifications to the research design and 
the outcomes for reporting will be discussed with practical examples from the UES. 
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Total Survey Error 


Total Survey Error (TSE) sits within a Total Survey Quality (TSQ) framework and provides 
an operational focus on survey design and execution. (Biemer & Lyberg, 2003). The TSE 
approach supports the identification and assessment of sources of error in the survey design, 
data collection, processing and analysis. TSE also encourages the consideration of these 
survey quality components within given design and budget parameters. The main criticism of 
TSE is that it is a theoretical, armchair exercise rather than a practical, statistical model of 
survey error (Groves & Lyberg, 2010). Despite this perceived shortcoming, TSE does offer a 
framework for a ‘theory of survey design’ that allows researchers to assess and prioritise key 
aspects of their research project (Groves, 1987). 

TSE is conceptualised as being composed of sampling error and non-sampling error. Errors 
of representation, sampling error, occur during the sample specification and the selection of 
the cases from the sample frame. Errors of measurement, non-sampling error is a broader 
concept encompassing systematic and random errors across, for example, the survey 
instrument, the respondent and the mode of data collection (McNabb, 2014). 

Table 1 summarises key research design components and the errors of representation that can 
occur in relation to each component (Biemer, 2010) (Biemer & Lyberg, 2003) (Blausis & 
Thiessen, 2012) (Groves, M, Crouper, Lepkowski, Singer, & Torangeau, 2009). Coverage 
error refers to mismatch between the population of interest and the population of the sample 
frame. Sampling error occurs when the incorrect people identified to participate in the survey. 
Non-response error is evident when sampling units do not participate in the survey or items 
are missing data. Adjustment error includes the use of incorrect or inappropriate weighting 
schema that are intended to correct for other errors of representation. 
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Table 1: TSE errors of representation in the context of the research design. 


Research design component 

Error of representation 

Target population 

Coverage error 

Sampling frame 

Sampling error 

Identified sample 

Non-response error 

Achieved interviews 

Adjustment error 


Minimising or managing survey error involves mediating a balance between a survey design 
of the required quality standard and a design that meets the available budget. This tension 
between quality and budget is often most evident when the survey involves a large scale data 
collection program or the population of interest is challenging to access. When assessing the 
error profile across all components of the survey cycle, researchers also need to be aware 
that: 

• addressing a particular component of may encroach on budget required for other 
mitigation activities (Blausis & Thiessen, 2012), 

• ‘upstream’ errors such as coverage error should be fixed before implementing non- 
response activities (Vicente & Reis, 2012) 

• minimising one source of error and increase another source of error. For example, 
increasing response rates could decrease representativeness (Hillygus, 2011). 

As explored in subsequent sections of this paper, errors of representation presented the 
greatest threat to data quality in the initial stages of the implementation of the UES. A 
detailed examination of TSE and the UES that includes a discussion of errors of measurement 
can be found in Whiteley (forthcoming). While adjustment errors and the associated 
weighting strategies to remediation these issues result in increased variance and as such can 
contribute to sampling error/errors of representation they were out of scope for our input to 
the 2013 UES and have not been explored as part of this paper. 
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The University Experience Survey 


With approximately 100,000 students participating each year, the UES is currently the largest 
national survey of higher education students. In 2013, the Social Research Centre and 
Graduate Careers Australia assumed responsibility for executing and improving the UES. The 
survey was originally developed and refined during 201 1 by a consortium commissioned by 
the Department of Education, Employment and Workplace Relations (DEEWR). 

The UES was designed to measure the engagement and satisfaction of current first and final 
year undergraduate students at higher education institutions. It is composed of a survey 
instrument, the University Experience Questionnaire (UEQ), and a survey methodology 
(Radloff, Coates, James, & Krause, 2011). With the initiation of the federal Quality 
Indicators for Learning and Teaching (QILT), the UES has become the core measure in a 
survey research program aimed at collecting feedback from undergraduate students, 
graduates and employers of graduates. As such, it is important to maximise the integrity of 
UES research design and execution prior to the introduction of associated quality indicators. 

A TSE approach was implemented to explicitly identify, explore and address the quality of 
the UES. 

TSE and the UES 

The UES can be broadly summarised as an online population survey of undergraduate 
university students. Representation error concerns relevant to online surveys tend to focus on 
coverage error and detail the shortcomings of non-probability panels that are typically used to 
source potential survey participants (Callegaro & Disogra, 2008), including the lack of 
representativeness of these panels (Blasius & Brandt, 2010) and the difficulties of 
generalising findings to the target population (Best, Krueger, Hubbard, & Smith, 2001) 
(Frippait & Marquis, 2010) particularly where potential respondents who are in-scope for the 
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survey are not part of the sample frame (Timmerman, 2002). Asan & Ayhan (2013) identified 
specific quality concerns relevant to sampling frames, or population lists, for online surveys 
and provided a series of weighting and adjustment procedures to correct for the resulting 
coverage errors. 

The original conceptualisation of the UES methodology suggested that this survey was a 
‘special case’ with the availability of a robust sampling frame and access to validated email 
addresses for all members of the population. Moving the UES towards this sampling frame 
that addressed all of the coverage error concerns associated with online surveys was a high 
priority for all implementations of the UES. Interestingly, as discussed in subsequent 
sections, the main threat to data quality was linked to over coverage rather than under 
coverage of the population. 

Debates around non-response error associated with online surveys comment on a range of 
issues such as lack of access to the internet and the extent to which those invited to complete 
the survey do not participate (Couper, 2000). Many of these discussions are located in 
commentary around multi-mode surveys where online surveys are implemented because they 
are cost effective but low response rates require the use of supplementary data collection 
approaches such as CATI to undertake non-response follow-up activities (de Leeuw, 2005). 
There are many suggestions in the research literature regarding the nature and timing of 
activities to improve response rates to online surveys including offering incentives, reminder 
emails and the way in which the questionnaire is presented to participants (Deutskens, 

Ruyter, Wetzels, & Oosterveld, 2004). Recent evidence suggests that retaining a single self- 
administered mode of data collection may improve data quality, largely due to the absence of 
error introduced by interviewer effects, however it is unclear whether this observation is 
directly linked to the topic of the survey (Nagelhout, Willemsen, Thompson, Fong, van den 
Putte, & de Vries, 2010). 
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Many of these issues relating to non-response and response maximisation for online surveys 
are directly relevant to the UES and were explored as part of the survey improvement 
activities in 2013 and 2014. Again, the UES is somewhat of a special case as the majority of 
the in-scope population are young students, which is a positive sign for response rates as 
youth, education level and current student status are associated with frequency of internet use 
and an increased likelihood of participating in online surveys (Vicente & Reis, 2013). Even 
so, to maximise the number of reportable data cells from the UES, it was necessary to 
implement a broad range of non-response activities as discussed in subsequent sections. 

All UES collections have been independently reviewed by Dennis Trewin, the former 
Australian Statistician, using a TSE approach. Summaries of the responses to the review 
feedback and the associated recommendations as well as the areas identified for improvement 
in 2013 are outlined in the following sections. 

Review of the 2011 UES 

As part of Trewin’ s 2011 review of the development of the UES (Appendix C, Radloff et al 
2011:pg 59), ten likely sources of error were identified including: 

• Poor questionnaire design 

• Survey frame coverage 

• Sample unrepresentative or inefficient (leading to inaccurate survey estimates) 

• Sample selection at risk of being manipulated 

• High non-response rate leading to non-response bias 

• Significant item level non-response 

• Coding errors and/or inconsistent coding 

• Inadequate data input validation checks 
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• Inadequate data output validation checks, and 

• Inappropriate estimation methods used. 

Of these potential survey errors, the unit (student) level non-response rate was identified as 
the source of greatest concern. The response rate to the pilot test was only 14 per cent and 
while this does not necessarily lead to non-response bias, it does substantially increase the 
risk that this could occur. 

Possible issues relating to coverage and sampling were also raised in the 2011 review of the 
UES. It was noted that individual universities provided a list of students that were thought to 
be in-scope for the UES and the sample was selected from these lists. Recommendations 
from the 2011 report highlighted specific concerns regarding the definition of the student 
population that was relevant to the UES. 

An additional aspect of the survey design that could be a source of error was that some 
institutions were responsible for sending out invitations to participate in the UES. This led to 
concerns that some universities may be selective regarding which students they choose to 
include or exclude in the survey. The review indicated that efforts should be made to 
independently confirm that all selected students were approached to participate. 

Overall, the 2011 review suggested that errors of representation were the primary areas for 
improvement in 2012. Errors of measurement, particularly those relating to the design of the 
questionnaire were regarded as having a negligible potential impact. 

Review of the 2012 UES 

The focus of Trewin’s 2012 review of the UES centred on the errors of representation that 
had (Appendix G, Radloff et al 2012) been identified by the 2011 implementation of the 
UES. It was noted that errors of measurement, including questionnaire design and validity, 
were not a focus for methodological improvement as their precieved risk to data quality was 
low. The 2012 UES review found that: 
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• Adequate checks were undertaken to ensure that the coverage of the student 
population was adequate 

• The survey design led to a sample that was as representative and efficient as it could 
be given some inherent limitations 

• The student level non-response rate was high but that mitigation activities should 
minimise resulting bias, and 

• Item level non-response was moderately high. 

A key change from the 2011 to the 2012 implementation of the UES was the introduction of 
Computer Aided Telephone Interviewing (CATI) as a non-response mitigation strategy. It is 
unclear from the 2012 UES Report why the online response rate was so low but it was within 
two percentage points of the 2011 pilot survey. Even with the supplementary telephone 
interviews, the overall response rate of 24 per cent was well below the national target of 35 
per cent. More importantly, there is little evidence to suggest that the telephone interviews 
improved representatives, particularly at an institution level as not all institutions were 
included in the CATI data collection program. 

Another difference between the 2011 and the 2012 surveys from a TSE perspective was the 
marked increase in the item-level non-response. With respect to the key items of interest, the 
UES scales, item level non-response to the online survey ranged from 7.3 per cent to 9.5 per 
cent. It was suggested that this did not effect the scale estimate but remains concerning given 
that item level non-response was so low in 201 1. 

The issue of in-scope population was not resolved in 2012 and the need to provide an 
accurate definition of ‘final year’ students was again identified. Sampling was again 
conducted centrally from files provided by each of the 40 universities based on the 
assumption that the frame was reasonably sound. A firmer recommendation for centralised 
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administration of the survey was made however it was acknowledged that some institutions 
needed to resolve privacy restrictions before this could be implemented. 

Mitigating errors of representation for the 2013 UES 

There were two main weaknesses of the 2011 and 2012 UES survey designs that underpinned 
the concerns regarding errors of representation. The key concern related to the absence of a 
clear operational definition of the in-scope population. This resulted in the creation of a 
sampling frame that was inconsistent across participating institutions and almost impossible 
to independently verify. As such, the focus of survey error mitigation activities for 2013 was 
aimed at ensuring that these two issues were adequately addressed. Coverage errors were 
regarded as being ‘upstream’ of other potential risks to data quality, such as sampling, which 
necessitated their prioritisation. 

In parallel with efforts to minimise coverage error, it was acknowledged that response rates 
needed to remain high and ensure as much as possible that the achieved interviews were 
representative of the in-scope population for the UES. As we were confident that the 
coverage errors would be adequately addressed before commencing fieldwork for 2013, 
additional strategies were put in place to mitigate survey and item-level non-response with a 
focus on maximising representativeness. Table 2 on the following page summarises the 
identified errors of representation and the range of strategies that were implemented. 
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Table 2. Summary of the identified errors of representation and the associated mitigation 


strategies implemented for the 2013 UES. 


Error of representation 

Mitigation strategy 

Sample frame not representative of 
the student population. 

Administrative data from the Higher Education 
Information Management System (HEIMS) data used 
to generate a national sample frame. 

In-scope population poorly defined 

Transparent definition of in-scope population created 
syntactically using established data elements from 
HEIMS. 

Ineligible cases included in the 
sampling frame 

Sample frame independently validated by institutions 
to ensure that ineligible cases are flagged and 
unavailable for selection. 

Survey non-response unacceptably 
high 

Collaborative relationships established with survey 
managers. Appropriate incentivisation scheme 
implemented. Response rates monitored and 
corrective action taken throughout fieldwork. 

Population sub-groups 
underrepresented 

Sample representativeness monitored and corrective 
action taken through targeted reminder emails and 
SMS’ throughout the fieldwork period. 

Item level non-response 
unacceptably high 

The input controls for each item part of the 
questionnaire logic programmed into the survey. 


Adapted from (Whiteley, forthcoming). 


The following sections detail the approaches undertaken to mitigate coverage error, survey 
non-response, population sub-group representation and item non-response for the 2013 UES. 

Coverage error 

A key challenge for the 2013 collection was to operationalise a technical definition of the in- 
scope population that was transparent and could be applied consistently across all 
universities. It was also decided to used a centralised rather than a decentralised approach to 
building the sample to maximise uniformity and also to minimise the burden on the survey 
managers (so they could be freed up to focus on other mitigation issues, including response 
maximisation). 
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In both 2011 and 2012, participating institutions provided extracts from their student systems 
to the contractor based on a narrative definition of the in-scope population resulting in a 
‘bottom-up’ approach to the creation of the sample frame. The UES reports indicate that the 
records were subsequently cleaned and coded in preparation for sample initiation. 

The 2013 implementation of the UES used population data from the Higher Education 
Information Management System (HEIMS) to create ‘top-down’ sample frames for 
individual universities. This approach supported a consistent sampling across all institutions 
and minimised the chance of bias being introduced into the sample selection process. 

Defining the in-scope population. HEIMS contains a data element that clearly 
identifies commencing students. As detailed in the following sections, the main challenge 
was to create a transparent operational definition of completing students. 

Commencing students. Commencing students were defined as first year students who 
were enrolled in an undergraduate course, studying onshore, commenced study in the relevant 
target year; and enrolled for at least one semester. In 2012, this definition was provided to 
participating institutions in 2012 and it was assumed that the correct records were extracted 
by each institution. It is unclear whether all universities used the relevant HEIMS data 
element or if they cleaned this data to exclude students that were ineligible to participate (see 
subsequent section on Identification of Ineligible Cases). 

Students conforming to the agreed definition of a commencing student were extracted from 
the national HEIMS Submission 1 Student File. Each university was asked to verify where 
possible, the active enrolment status of the students selected into the sample. 

Final year students. Students in their final year of study are narratively defined as 
being enrolled in an undergraduate course, generally in their third year of study, and studying 
onshore. Each university was required to extract relevant student records from their 
administrative systems based on this loose definition. As noted previously, this definition 
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proved to be problematic for institutions resulting in confusion regarding which ‘final year’ 
students were in-scope (Radloff et al, 2012). 

The lack of clarity regarding the definition of final year students seems to have resulted in the 
slightly unusual situation of over coverage. In 2012 445, 332 students were identified as 
being in-scope for the survey compared with only 341, 343 in 2013. The 2012 figure cannot 
be independently verified and it is likely that the definitional confusion led to the inclusion of 
approximately 100,000 middle years students in the sample. 

Two options for operationalising the definition of final year students were investigated with a 
correction for course duration being the main difference. As this option adjusting for course 
length appeared to appropriately identify the majority of completing students for most 
institutions, this approach was implemented for the 2013 collection. In a small number of 
cases, an adjustment was made for specific institutions to take into account subject offerings 
outside the standard ‘two semester’ structure. 

Identification of ineligible cases. Submission 1 HEIMS data is not formally validated 
and has not necessarily been subjected to the same audit and cleaning procedures as the 
Submission 2 file. An initial examination of the file revealed that there were marked 
differences between institutions in terms of their mid-year intakes, attrition rates, mid-year 
completions and internal course transfers. To maximise the quality of the sample data, 
institutions were asked to inspect the selections and update a small set of key variables. 

As part of the verification process, institutions were provided with the opportunity to identify 
students that were out of scope for the UES. Most of the small number of exclusions related 
to course deferrals and enrolments in post-graduate courses. In the majority of cases, the 
universities chose to retain all of the students that had been selected and there was no 
indication at all in either 2013 or 2014 that cases were being excluded in appropriately. 
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Non-response error 


As identified in the 2012 and 2013 reviews of the UES, unit non-response and 
representativeness was two areas clearly identified for future improvement. As part of the 
2013 review, there appeared to be emerging concerns regarding item-level non-response. 
Survey and item- level non-response mitigation strategies are discussed below. 

Unit level non-response. In keeping with the UES 2011 review recommendations, the 
main focus for fieldwork activities was on achieving the overall response rate of 35 per cent. 
This target was regarded as a stretch but likely to be achievable (Radloff, et al 2011). 
Disappointingly, the online response rate was similar to the 2011 pilot test at 20.2 per cent 
and select telephone interviews were required to ‘top-up’ underperfonning institutions. While 
the introduction of an additional data collection mode had the potential to decrease non- 
response bias, there was also the possibility that it could increase measurement error through 
mode and interviewer effects. 

A key aim of the 2013 response maximisation strategy was to ensure that the number of 
interviews achieved was sufficiently high to ensure that telephone non-response follow-up 
would not be necessary. With the support of the institutional survey managers, a ‘whole-of- 
UES strategy was deployed that included the use of a prize draw incentive, hard-copy letters 
to non-responders and targeted email reminders. A prize draw was conducted for each 
institution, typically within two to three weeks of survey initiation to encourage students to 
complete the UES as soon as possible. 

The non-response strategy implement for 2013 resulted in a significant increase in the overall 
response rate with a national increase of almost nine percentage points. All institutions, with 
the exception of two universities that were ‘high-performers’ in 2012, showed substantial 
response rate improvements of up to 26 percentage points. The largest reduction in response 
rate was 0.3 percentage points. 
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Table 3. Institutional, online response rates to the 2012 and 2013 UES. 



2013 

2012 

Change 

Australian Catholic University 

23.7 

11.6 

12.1 

Bond University 

32.8 

6.7 

26.1 

Central Queensland University 

36.0 

25.3 

10.7 

Charles Darwin University 

40.5 

25.7 

14.8 

Charles Sturt University 

32.3 

21.7 

10.6 

Curtin University of Technology 

26.1 

23.8 

2.3 

Deakin University 

29.2 

14.7 

14.5 

Edith Cowan University 

29.3 

25.7 

3.6 

Flinders University 

35.2 

21.1 

14.1 

Griffith University 

23.5 

19.5 

4.0 

James Cook University 

29.0 

19.1 

9.9 

La Trobe University 

33.0 

20.7 

12.3 

Macquarie University 

26.3 

18.9 

7.4 

MCD University of Divinity 

50.5 

44.6 

5.9 

Monash University 

39.7 

23.3 

16.4 

Murdoch University 

30.6 

20.1 

10.5 

Queensland University of Technology 

29.4 

20.8 

8.6 

RMIT University 

20.8 

3.2 

17.6 

Southern Cross University 

24.4 

15.3 

9.1 

Swinburne University of Technology 

25.5 

13.2 

12.3 

The Australian National University 

29.3 

29.6 

-0.3 

The University of Adelaide 

41.4 

24.6 

16.8 

The University of Melbourne 

34.5 

22.0 

12.5 

The University of Notre Dame 

26.0 

17.1 

8.9 

The University of Queensland 

32.5 

24.9 

7.6 

The University of Sydney 

30.3 

23.1 

7.2 

The University of Western Australia 

39.7 

39.8 

-0.1 

University of Ballarat 

22.1 

20.4 

1.7 

University of Canberra 

24.4 

19.8 

4.6 

University of New England 

32.9 

16.3 

16.6 

University of New South Wales 

27.0 

17.5 

9.5 

University of Newcastle 

34.0 

30.9 

3.1 

University of South Australia 

25.2 

23.6 

1.6 

University of Southern Queensland 

25.2 

15.7 

9.5 

University of Tasmania 

33.0 

22.7 

10.3 

University of Technology Sydney 

28.2 

13.4 

14.8 

University of the Sunshine Coast 

29.2 

23.5 

5.7 

University of Western Sydney 

26.6 

22.2 

4.4 

University of Wollongong 

23.5 

20.1 

3.4 

Victoria University 

17.9 

10.4 

7.5 

Total 

29.3 

20.2 

9.1 
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Population sub-group representation. It is apparent from Table 4 that there are 
number of sample parameters closely match the achieved respondent profile in 2013. Status, 
course of study, course of study type, ATSI status, and type of attendance are similar for both 
sample members and survey respondents, a finding that was consistent with the 2012 UES 
(Radloff, et al 2012). 

Table 4. Sample and respondent characteristics for the 2013 UES. 



Sample 

% 

Respondents 

% 

Base 

344,692 


100,225 


Status 

Commencing 

208,307 

60.4 

59,653 

59.5 

Final year 

136,385 

39.6 

40,572 

40.5 

Gender 

Male 

148,264 

43.0 

33,349 

33.3 

Female 

196,428 

57.0 

66,876 

66.7 

Combined course of study indicator 

Combined/double degree 

37,887 

11.0 

11,919 

11.9 

Single degree 

306,805 

89.0 

88,306 

88.1 

Course of study type 

Bachelors Graduate Entry 

4,925 

1.4 

1,627 

1.6 

Bachelors Honours 

10,096 

2.9 

3,921 

3.9 

Bachelors Pass 

320,155 

92.9 

92,808 

92.6 

Associate degree 

4,959 

1.4 

908 

0.9 

Advanced Diploma 

1,494 

0.4 

408 

0.4 

Diploma 

2,811 

0.8 

495 

0.5 

Other undergraduate award 

252 

0.1 

58 

0.1 

Aboriginal and Torres Strait Islander 

Indigenous 

4,126 

1.2 

1,067 

1.1 

Non-Indigenous 

334,617 

97.1 

97,535 

97.3 

Not stated 

5,949 

1.7 

1,623 

1.6 

Type of attendance code 

Full-time 

307,739 

89.3 

90,137 

89.9 

Part-time 

36,953 

10.7 

10,088 

10.1 

Language spoken at home code 

English 

258,416 

75.0 

77,208 

77.0 

Language other than English 

81,537 

23.7 

21,931 

21.9 

Not stated 

4,739 

1.4 

1,086 

1.1 

Citizen/resident indicator 

Domestic 

294,662 

85.5 

88,067 

87.9 

International 

50,030 

14.5 

12,158 

12.1 
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Gender bias is still clearly apparent in the 2013 UES with a substantially lower response rate 
for males, a finding that was also observed in relation to the 2012 collection. For the 2012 
survey, the proportion of male online respondents was 35.1%, similar to the 33.3% achieved 
in 2013. It should be noted that the significant increase in response rate from between 2012 
and 2013 did not proportionally increase the level of gender bias which was a positive 
outcome. 

Item level non-response. As the 2012 UEQ was administered in two modes, CATI and 
online, only the online component of the 2012 collection was compared to the 2013 data. In 
2012, the average item-level non-response increased substantially from the pilot test of the 
UES to 7.7 per cent. For the 2013 survey, average item non-response was reduced 
significantly to 1.0 per cent. 


An inspection of some of the items with higher levels of non-response suggests that the 
question wording may be contributing to students refusing or being unable to answer. Using 
the acdavail (1.4 per cent non-response) and acdhelp (1.5 per cent non-response) items as 
examples it is evident that the question stems and response frames are not as harmonious as 
they could be. 

acdintro During 2013, to what extent have you found academic or learning advisors to 

be... 

STATEMENTS 

acdavail Available? 

acdhelp Helpful? 

RESPONSE FRAME 

1 . Had no contact 

2. Not at all 

3. Very little 

4. Some 

5. Quite a bit 

6. Very much 
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Looking at these question stems and the response frame, it is unclear what ‘quite a bit 
available’ or ‘some helpful’ actually means. It would be useful to cognitively test these items 
or a refreshed response frame prior to the next implementation of the UES. Revising the 
UES in 2013 or 2014 was not possible due to timeframe constraints. 


Mitigating errors of representation for the 2014 UES 

Sampling error 

As there were so many concerns about the quality of the sample frame and the accuracy of 
the sample selection from the 2012 collection, it was decided to address coverage issues 
before remediating the sampling errors. Table 5 summaries the sampling errors and the 
mitigation activities that were undertaken in 2014. These sampling errors and the relevant 
strategies are interdependent and, as such, a combined discussion of sample size, margin of 
error and anticipated response rate is provided in the following section. 

Table 5. Summary of the identified sampling errors and mitigation strategies implemented 


for the 2014 UES. 


Sampling errors 

Mitigation strategy 

Sample size inappropriate 

Implement an approach that supports accurate 
sampling at a strata level (rather than nationally) 

Margin of error higher than 
expected 

Moderate expectations regarding precision of 
estimates 


(Whiteley, forthcoming). 

In 2012 and 2013, a similar approach was used to identify the appropriate sample size for the 
UES collections. Strata were created based on 45 Study Areas and current status 
(commencing or completing). Based on the sampling rules in the 2012 UES National Report, 
strata with up to 1,333 in-scope students were a census and a random selection of 1,333 
students was taken from lager strata. The logic behind the selection approach was that a 
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sample of 1,333 would yield at least 200 completed interviews and these 200 interviews 
would ensure that a confidence interval of ±5 per cent at a 95 per cent level of confidence. 

It was apparent during the set up and fieldwork phases of the 2013 UES that the method used 
to determine sample size was less than optimal. Specifically, this approach did not take into 
account the wildly differing response rates across strata which led to the substantial 
oversampling of large strata resulting in: 

• achievement rates of more than 200 completed interviews 

• over representation of these groups, and 

• an increased gender imbalance. 

Based on observations during the 2013 fieldwork, it was evident that the large strata tended to 
be in Study Areas that included female dominated courses such as nursing and teaching. As 
females are generally more likely participate in surveys the naturally higher response rate and 
absence of quotas for strata resulted in an excess of female participants. 

For the 2014 UES collection, a two stage approach was undertaken to determining sample 
size and the number of student records that would need to be selected to achieve these targets. 
Sample sizes were calculated at the strata level taking into account the number of records 
available for sampling and the requirement to report data at a 90% confidence level, +/- 5%. 

A finite population correction was also applied to each stratum. 

When the response rates required to achieve the number of completed surveys for each 
stratum were analysed, it was apparent for a large proportion of strata that the necessary 
targets were unachievable based on the known response rates to the 2012 survey. The key 
driver of the need for extremely high response rates resulted from the fact that a student 
numbers were comparatively small in most subject areas for many institutions. Using the 
revised 2014 approach to sampling, almost all strata required a census and in many cases a 
response rate of between 70 and 100 per cent was required. 
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In consultation with the department, the level of reporting precision was modified to a 90% 


confidence level +/- 7.5%. In almost all instances, the number of records that needed to be 
sampled was retained but the required response rate was lowered to a level that was more 
achievable. It was still the intention of the operational team to aim for a 5% confidence 
interval and this was used as a ‘background target’ with a view to making this the actual 
target in future implementations of the UES. 

As was the case in 2013, selected sampled was checked against the population to confirm that 
gender, qualification, mode of attendance, broad field of education and citizenship 
characteristics were appropriately represented in the sample. 


Overall quality improvement & cost effectiveness 

The approach to conducting the UES for the 2013 and 2014 cycles was based on a careful 
consideration of potential sources of survey error tempered by an appreciation of the 
compressed timeline for both cycles of data collection. The TSE framework provided a: 

• conceptual framework for evaluating the design of the UES, 

• structured approach to making decisions about modifying the UES to support 
continuous improvement, 

• method for determining an optimal research design that offered good value for 
money, 

• way to prioritise survey errors for mitigation, and a 

• means to challenge accepted paradigms regarding response rate as the primary 
indicator of a ‘good’ survey. 

The survey framework implemented for the 2013 and 2014 UES was designed to address 
errors of representation that had been identified as part of the previous methodological 
reviews. More specifically: 
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• Using HEIMS, creating an effective operational definition of ‘completing’ students 
and a transparent sample selection validation process reduced coverage errors. 

• A more detailed sampling strategy and close monitoring of study area targets within 
institutions decreased sampling errors. 

• Increased unit level response rates and a reduction in item-level non-response 
contributed to a lessening of non-response errors. 

Overall the greatest reduction in error was realised by addressing the population over 
coverage and it appears to be the case that while there are some small improvements that can 
be made in relation to the definition of students in their final year, there is little to be gained 
from an ongoing focus on coverage error. There is still evidence of sampling and non- 
response issues that need to be resolved and these areas have been targeted for ongoing 
attention prior to the 2015 UES. It is unlikely that the sampling issues will be fully resolved 
in the short term due to the requirement for very high response rates from students enrolled at 
smaller institutions however continuing to use a TSE framework will ensure that all relevant 
avenues are explored within budget. 

As suggested previously, a TSE approach usefully incorporates an appreciation of cost 
constraints in relation to achieving quality survey outputs. Based on our analysis of the 2013 
and 2014 UES collections, the cost of implementing the improvements was budget neutral. 
Mitigating non-response error, and mode effects, by improving online response rates through 
the use of hard-copy letters to individual students was similar to the cost incurred for the 
telephone non-response follow up phase of the 2012 UES. Coverage error and sampling error 
were minimised through the effective implementation of strategies to use existing 
administrative data, such as HEIMS, and resources to ensure a positive outcome. 

The observation that the theoretical TSE framework doesn’t provide unambiguous answers to 
questions about an optimal research design is correct but TSE does support a logical approach 
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to making evidence-based decisions about trade offs between survey quality and cost. 
Thinking about research execution from the perspective of TSE also provides researchers 
with an opportunity to reflect on their research practice and identify areas for improvement. 
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